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BACKGROUND OF THE INVENTION 



10 1. Field of the Invention 

The present invention relates to methods and apparatus for synchronizing 
and propagating distributed routing databases. The invention also relates to 
methods for distributing routing data within a distributed processor router 
15 system. 

2. Background of the Related Art 

In the context of internetworking, routing is the coordinated transfer 
20 of information from a source to a destination via hardware known as a 
router. Routing occurs at Layer 3, the network layer, of the OSI reference 
model of the ISO (International Society for Standardization). The OSI 
reference model is a conceptual model composed of seven layers, each 
specifying particular network ftmctions. The two lowest layers (layers 1 and 
25 2) of the OSI model, namely the physical and data link layers, are 

implemented in both hardware and software. Layer 3 and layers upwards 
therefrom are generally implemented only in software. 
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Using terminology of the International Organization for 
Standardization (ISO), network devices may be classified as follows. Those 
devices with the capability to forward packets between subnetworks are 
referred to as intermediate systems (ISs). (In contrast, network devices 

5 without such capabilities are called end systems). Intermediate systems may 
be classified as intradomain ISs, i.e., those which can communicate within 
routing domains, and interdomain ISs which can communicate both within 
and between routing domains. A routing domain, or autonomous system, 
can be considered to be a part of an internetwork which is regulated under 

1 0 common administrative authority, 

A key component of routing is determination of optimal routing 
paths for data packets. Thereafter a second component, which may be 
referred to as "forwarding", comprises transporting packets through the 

15 internetwork. Determination of optimal routing paths relies on one or more 
routing protocols to provide and update a routing database for each router in 
a network. Depending on the particular routing protocol(s) used, various 
metrics are involved in building the routing database. Metrics that may be 
used by various routing protocols, either singly or as components of hybrid 

20 metrics, include: bandwidth, cost, path length, reliability, and load. Such 
metrics are well known in the art. 

Routing protocols are used to determine best routes for transporting 
packets through an internetwork. Routing in a network can be classified as 
25 either dynamic or static. Static routing is accomplished by using table 

mappings which are entered by a user (e.g. network administrator) prior to 
routing, and are only changed by user input. Dynamic routing is 
accomplished by routing protocols that adjust to changing network 
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conditions in response to incoming route update information. As a result, 
routes are recalculated, new routing update messages are sent out to peer 
routers, and updated routing databases are constructed. Routing protocols 
may be interior or exterior. Conventionally, interior routing protocols are 

5 used for determining routes within a routing domain. Examples of interior 
routing protocols are Routing Information Protocol (RIP) and Open 
Shortest Path First (OSPF). Exterior routing protocols exchange routing 
information between routing domains. Examples of exterior routing 
protocols are Border Gateway Protocol (BGP) and Exterior Gateway 

10 Protocol (EGP). 

OSPF is a unicast routing protocol that requires each router in a 
network to be aware of all available links in the network, OSPF calculates 
routes from each router running the protocol to all possible destinations in 
15 the network. Intermediate System to Intermediate System (IS-IS) is an OSI 
link-state hierarchical routing protocol based on DECnet Phase V routing, 
whereby ISs (routers) exchange routing information based on a single 
metric, to determine network topology. 

20 BGP performs interdomain routing in TCP/IP networks. As an 

exterior gateway protocol (EGP), BGP performs routing between multiple 
routing domains and exchanges routing and reachability information with 
other BGP systems. Each BGP router maintains a routing database that lists 
all feasible paths to a particular network. The router does not refresh the 

25 routing database, however. Instead, routing information received from peer 
routers is retained until an incremental update is received. BGP devices 
exchange routing information upon initial data exchange and after 
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incremental updates. When a router first connects to the network, BGP 
routers exchange their entire BGP routing tables. 

In order to update their routing databases, routers send and receive 
5 information regarding network topology. Examples of such information 
include routing update messages, and link-state advertisements. By 
communicating with other routers in this way, each router obtains a routing 
database that defines the current topology of the network of which it is a 
part, enabling determination of optimal routing path. 

10 

Entries are added to and removed from the route database either by 
the user (e.g., a network administrator) in the form of static routes, or by 
various dynamic routing protocol tasks. In dynamic routing, routes are 
updated by software running in the router. The routing database defines a 
15 mapping from destination address to logical (output) interface, enabling the 
router to forward packets along the best route toward their destination. The 
route database is also the principal medium used to share routes among 
multiple active routing protocols. Thus, the routing database comprises an 
essential entity at the heart of every router. 

20 

Typically, two or three routing protocols may be active in any one 
router. The routing database as such is a superset of the set of routes 
actually used for forwarding packets. This is due, in part, to the fact that 
different routing protocols compute their preferred routes independently of 
25 each other, based on different metrics. Only when all route entries generated 
by the full complement of routing protocols are shared in the routing 
database, or route table, can the best routes be selected. The result of this 
selection is a subset of the routing database conmionly referred to as the 
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forwarding table. The forwarding table can be considered a filtered view of 
the routing database. The forwarding table is used by all entities of the 
router that have to forward packets in and out of the router. 

5 In conventional or prior art non-scalable routers, which have a 

modest number of interfaces, there is a single copy of the routing database 
shared by all of the routing protocols. In non-scalable routers, the 
computational power available to the routing protocols is typically limited to 
a single processor. Also, in non-scalable routers, the number of entities 

10 requiring a copy of the forwarding table is relatively small 

In contrast, in routers with a relatively large number of interfaces, a 
possibility exists for imposing much higher computational loads on the 
processor, up to a point where it is no longer feasible to run all routing 

15 protocols on the same processor. In order to realize improved performance 
fi*om such routers, the protocol computational load must be distributed onto 
a plurality of processors. Furthermore, in routers with a very large number 
of interfaces, the number of entities requiring a copy of the forwarding table 
can be very large, for example, numbering several thousands. This latter 

20 situation also imposes higher computational loads and the need for a 
plurality of processors per router. 

However, running the routing protocols on a plurality of processors, 
each processor having a copy of the routing database, introduces a potential 
25 problem into the routing system. The problem is the critical requirement to 
keep all copies of the routing database consistent. This requirement is critical 
because the view of the routing database presented to the routing protocols 
is vital to correct routing. Moreover, the ability to provide an accurate and 
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timely copy of the forwarding table to a very large number of entities in the 
system is necessary in order to leverage the benefits provided by a 
distributed routing database environment. 
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The instant invention provides a method for the distribution and 
synchronization of the routing database and forwarding table to a large 
number of entities within a distributed processor environment of a scalable 
router. 



SUMMARY OF THE INVENTION 



According to one aspect of the invention, there is provided a method 
for the synchronized distribution of routing data in a distributed processor 

15 router. The invention allows multiple routing databases, formed by 

distributed routing protocols, to be synchronized. The invention further 
allows distributed propagation of the synchronized database. 

One feature of the invention is that it enables the scaling of routing 
protocol tasks instantiated on multiple processors. Another feature of the 

20 invention is that it provides a distributed processor router environment, in 
which a plurality of processors host at least one of a plurality of different 
routing protocols. Another feature of the invention is that it provides a 
route table manager for controlling the propagation of a synchronized 
routing database within a distributed processor environment. 

25 

One advantage of the invention is that it allows routing databases to 
be constructed and propagated in a distributed manner by instantiating 
routing protocol tasks on multiple processors. Another advantage of the 
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invention is that it provides a method for exchanging route data between a 
plurality of processors within a distributed processor router environment, 
wherein the exchange of route data is controlled by a route table manager 
(RTM). Another advantage of the invention is that it provides a method for 

5 registering a first RTM task as a client of a second RTM task in order to 
establish a first RTM task-second RTM task client-server relationship, 
wherein the first RTM task and the second RTM task occupy either the same 
hierarchical level or different hierarchical levels. Another advantage of the 
invention is that it provides a method for estabhshing a first RTM task- 

10 second RTM task client-server relationship, wherein the first RTM task is 
running on a line card of a highly scalable router, and the second RTM task 
is running on a control card of the same router. 

These and other advantages and features are accompUshed by the 
15 provision of a method of synchronized distribution of routing data in a 
distributed processor router, including the following steps: a) running zero 
or more routing protocols of a complement of routing protocols on each of a 
first plurality of processors, wherein each routing protocol of the 
complement of routing protocols generates routing data; b) registering each 
20 of the first plurality of processors with at least one other of the first plurality 
of processors; c) exchanging the routing data between members of the first 
plurality of processors, such that each of the first plurality of processors 
receives a flail complement of routing data generated by the complement of 
routing protocols, the complement of routing data providing a complete 
25 routing database; d) forming a forwarding database fi-om the complete 
routing database, the forwarding database comprising a subset of the 
complete routing database; and e) propagating the forwarding database from 
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the first plurality of processors to a second plurality of processors, wherein 
the second plurality of processors are not running a routing protocol. 

These and other advantages and features are accomplished by the 
5 provision of a method of distributing routing data in a distributed processor 
router under the control of a route table manager (RTM), the router running 
a complement of routing protocols and the router having a plurality of 
control cards and a plurality of line cards, wherein this method includes: a) 
running an RTM task on each of the plurality of control cards, and running 
10 an RTM task on each of the plurality of line cards; b) generating routing data 
on at least one of the plurality of control cards; c) under the control of the 
RTM, distributing at least a portion of the routing data from at least one of 
the plurality of control cards to at least one other of the plurality of control 
cards; and d) again under the control of the RTM, distributing at least a 
1 5 portion of the routing data from at least one of the plurality of control cards 
to at least one of the plurality of line cards. 

These and other advantages and features are accomphshed by the 
provision of a method of registering a route table manager (RTM) task client 

20 with a RTM task server within a distributed processor router, including the 
steps of a) querying, via the RTM, a location service for a first node list, 
wherein the location service is a directory listing the location and status of 
each task running within the router, and wherein the first node list comprises 
a list of prospective RTM task servers; b) sending the first node list from the 

25 location service to a would-be RTM task client; c) selecting, by the would- 
be client, a first node firom the first node list; d) sending a registration 
request fi*om the would-be client to the first node selected in said step c); e) 
adding the would-be chent as a client of the first node selected, whereby the 
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first node selected is a server of the client; and f) after step e), sending a 
registration response from the server of the client to the client. 

These and other advantages and features are accomplished by the 
5 provision of a distributed processor router, including: a plurality of control 
cards, and a plurality of line cards, the plurality of control cards having a first 
plurality of processors, wherein each of the first plurality of processors runs 
zero or more routing protocols of a complement of routing protocols, each 
routing protocol of the complement of routing protocols generates routing 
10 data, and each of the first plurality of processors registers with at least one 
other of the first plurality of processors for exchange of the routing data 
between members of the first plurality of processors such that each of the 
first plurality of processors receives a Ml complement of routing data. 

1 5 These and other advantages and features of the invention will be set forth in 

part in the description which follows and in part will become apparent to those 
having ordinary skill in the art upon examination of the following, or may be learned 
from practice of the invention. The advantages of the invention may be realized and 
attained as particularly pointed out in the appended claims. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Figs. lA, IB and IC are block diagrams showing basic architecture of a 

scalable router according to an embodiment of the invention; 
5 Fig. 2 schematically represents exchange of route data generated by 

different routing protocols, according to an embodiment of the invention. 
Fig. 3 schematically represents exchange of route data generated by 

two different routing protocols showing four servers and two clients, 

according to an embodiment of the invention. 
10 Fig. 4 schematically represents chronology of RTM-mediated data 

flow between two control cards, according to one embodiment of the 

invention.; 

Fig. 5 schematically represents a hierarchical relationship of RTM 
tasks according to a preferred embodiment of the invention. 
15 Fig, 6 schematically represents a hierarchical relationship between 

route table manager tasks, according to one embodiment of the invention; 

Fig. 7A schematically represents the distribution of route data from a 
route table manager Level- 1 task primary server to a route table manager 
Level-2 task client, according to the invention; 
20 Fig. 7B schematically represents the distribution of route data from a 

route table manager Level- 1 task secondary server to a route table manager 
Level-2 task client, according to an embodiment of the invention; and 

Fig. 8 schematically represents a series of steps involved in a method 
for synchronized distribution of routing data within a distributed processor 
25 router, according to another embodiment of the invention. 
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ACRONYMS 



The following acronyms and abbreviations are used in the description which follows: 

BGP: Border gateway Protocol 
5 CCB : Client Control Block 

IS-IS: Intermediate System to Intermediate System 

ITC: Inter task communication 

LS: Location Service 

OSPF: Open Shortest Path First 
10 RTM: route table manager (also referred to as the global route table 

manager (GRTM)). 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

15 

In order to place the invention in perspective for the better understanding 
thereof, there now follows, with reference to Figs. 1 A-IC, a brief description of 
a scalable router which may be used in conjunction with the instant invention. 
Fig. lA is a block diagram showing the basic architecture of a router 10. Each 

20 router 10 may include a plurality of shelves, represented in Fig. lA as 20 A to 
20N. As shown in Fig. IB, each shelf 20 can include a plurality of line cards, 
represented as 40A to 40N. For the purpose of clarity, only two control cards 
are shown in Fig. IB; however, it is to be understood that in practice larger 
numbers of control cards can be used according to the invention. Each control 

25 card 30 is in communication with at least one line card 40. For example, control 
card 3 OA is shown as being in communication with line cards 40 A and 40N on 
shelf 20A. Again, for the purpose of clarity, only two line cards are shown as 
being in communication with control card 3 OA. However, according to the 
invention, larger numbers of line cards may be connected to each control card. 
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Fig. IC shows line card 40, which could be any of the hne cards from 
a shelf of router 10. Line card 40 has a plurality of ports, or exterior 
interfaces, 50A, 50B, through 50N connected thereto. Although, only three 
5 interfaces are depicted in Fig. IC, it is to be understood that a much larger 
number of interfaces may be used in practice. 

Introduction to a Route Table Manager 

1 0 A route table manager (RTM) of the instant invention is a 

multifaceted software suite having a plurality of functions (tasks) that 

include, but are not necessarily limited to, the following: 

1 . messaging between RTM task servers and RTM task clients to 

form scalable and fault tolerant distribution topologies; 
15 2. managing exchange of database information between RTM tasks 

running on separate processors within a distributed processor environment; 

3. constructing a routing database from the sum of database 
information a) generated locally by tasks running on a local processor, and b) 
generated by and received from tasks running on at least one remote 

20 processor; 

4. constructing a forwarding database from the routing database; and 

5. propagating the forwarding database from RTM tasks having a 
higher hierarchical level (Level-l tasks) to RTM tasks having a lower 
hierarchical level (Level-2 and lower-level tasks). 

25 

In a distributed multi-processor router, such as is encountered 
according to certain aspects of the instant invention, the RTM distributes 
information on dynamic routes, static routes, and interface information, 
hereafter referred to as database information. In return, RTM relies on a 
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number of tasks (services) for database information updates. Such tasks 
include those of dynamic routing protocols, IP, and an interface manager 
task. Routing protocols provide the RTM with updates on dynamic routes. 
IP tasks provide the RTM with updates on static routes. The interface 
5 manager task manages the ports, or external interfaces, of the router system, 
and provides the RTM with interface information. Interface information 
relates to a specific interface from which to dispatch a particular packet. 
Interface information, in general, is well known in the art. 

10 The sum of the database information provided by services is 

collectively referred to as the routing database. Route entries maintained in 
the routing database include best and non-best routes. For example, all route 
entries that were injected by different routing protocols of the system's 
complement of routing protocols are stored in the routing database. 

15 However, for a pluraUty of entries having the same destination prefix, only 
one of the entries is deemed the best. The decision as to which of those is the 
best entry (i.e. the best route for forwarding a packet) is based on a pre- 
configured preference value assigned to each routing protocol. For example, 
if static routes have a high preference value and IS-IS routes have a low 

20 preference value, and a route entry having the same destination prefix was 
injected by each protocol, although both entries will remain in the routing 
database, the static route is considered to be the best route. In embodiments 
of the invention, both the best routes and the non-best routes, as well as 
interface information, are retained in the routing database. A subset of the 

25 routing database exists which is referred to as the forwarding table. The 
forwarding table contains all route entries that are deemed the best plus all 
interface information. Therefore, according to the invention, both the best 
routes and the interface information define the forwarding table. 
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A task of the RTM software suite typically runs on each of the 
plurality of processors of a multi-processor scalable system, including 
processors on control cards and line cards. The RTM task executing on each 

5 processor can be classified as either a Level- 1 RTM task (LI) or a Level-2 
RTM task (L2), and the processor may be termed an LI or an L2 as a result. 
The distinction between an LI and an L2 is in general the presence of either 
a routing database or a forwarding table. An LI RTM task maintains the 
routing database and an L2 RTM task maintains the forwarding table. A 

1 0 subset of the plurality of processors of the system is statically configured to 
host an LI RTM task and is referred to as the LI pool. All other processors 
of the system outside of the LI pool host an L2 RTM task. 

As previously described, the RTM depends on a number of services 
for updates in routing information. A processor within the LI pool may be 

1 5 running a number of such services, or none at all. Examples of such services 
include the IP routing protocols, OSPF, BGP, integrated ISIS, etc. (See, for 
example, C. Huitema, Routing in the Internet, 2"^ Edition, Prentice Hall 
PTR, 2000.) According to the invention, each LI is responsible for 
constructing a routing database from mformation generated in part by tiie 

20 local service(s), and in part from information generated by services running 
in association with otiier Lis. To obtain information that is generated by 
non-local services, i.e. information generated by services running on otiier 
Lis, an LI must register with at least one otiier LI where tiie service is 
running. According to the invention, in order to efficientiy exchange 

25 locally generated information between Lis, each LI can register witii at 
least one otiier LI as needed, on a per-service basis, to receive updates on 
the full complement of route data which is generated non-locally. 
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Lis register with each other for distribution of the following types of 
database information: dynamic routes including best and non-best routes, 
static routes including best and non-best routes, and interface information. 
An LI is classified as an LI server or LI client for a given type of database 
5 information, depending on the existence of local services. An LI task is an 
LI server for a particular type of database information if the service which 
generates that information is running locally. An LI task is an LI client for a 
particular type of database information if the service which generates that 
information is not running locally and the LI task has registered with an LI 

10 server for information of this type. For example, if a BGP task was running 
on a given processor, the LI task on that processor is considered an LI 
server for BGP route information. If the same LI task has registered with a 
remote LI task for OSPF route information, the former LI task is 
considered an LI client of the remote LI task with regard to OSPF route 

15 information. 



Fig. 2 schematically represents exchange of route data, generated by 
different routing protocols, between a plurality of control cards 30A, SOB, and 
3 ON within a distributed processor, scalable router, according to one 

20 embodiment of the invention. As alluded to hereinabove, the inventors have 
determined that superior performance from a scalable router is attained when 
routing protocols are distributed among control cards of the router. That is, 
superior performance is attained by running a plurality of different routing 
protocols on a plurality of processors within the control plane (on control cards) 

25 within the router. According to one embodiment, each of the plurality of 

processors is situated on a different control card of the router. With reference to 
Fig. 2, the plurality of control cards is represented by control cards 3 OA, 3 OB, 
and SON. In the example shown in Fig. 2 a service or routing protocol task runs 
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on each control card 3 OA, SOB, 3 ON. Therefore, according to the definitions 
presented hereinabove, a Level- 1 task (LI) of the RTM is running on each 
processor. In particular, according to the example shown in Fig. 2, control 
cards 30A, 30B, 30N run routing protocol A, routing protocol B, and routing 

5 protocol N, respectively. Routing protocol A, routing protocol B, and routing 
protocol N, provide route data A, route data B, and route data N, respectively. 
As described hereinabove, the LI for each control card requires route data from 
the full complement of routing protocols running on the plurality of control 
cards 30A, 30B, and 30N. Lis therefore exchange route data by registering 

10 with other LI s on a per-service basis. 

Fig. 3 schematically represents exchange of route data generated by two 
different routing protocols showing four servers and two clients, according to an 
embodiment of the invention. This aspect of the instant invention relates to the 
registration of Lis with at least one other LI, on a per-service basis, for the 

15 facile exchange of non-locally generated route data. Each entity I-IV represents 
an LI task: LI A, LI A, LIB, and LIB', respectively. For the purpose of this 
example, the routing protocol tasks are designated as routing protocol A (RPA) 
in the case of LI A and LI A, and routing protocol B (RPB) in the case of LIB 
and LIB'. Under the control of the RTM, LIA registers as a client with both 

20 LIB and LIB' for information generated by routing protocol B, wherein both 
LIB and LIB' are servers. Similarly, LIB' registers as a client with both LIA 
and LIA for information generated by routing protocol A, wherein both LIA 
and LIA are servers. Thus, the same entity may have both client and server 
functionality concurrently. For the sake of clarity, LIA and LIB are not shown 

25 as clients, but as servers only, therefore sending, rather than receiving 
information. 

In the arrangement shown in Fig. 3 LI A is registered with both LIB and 
LIB', which both run RPB, and LIB' is registered with both LIA and LIA, 
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which both run RPA. This redundancy in preferred embodiments of the 
invention provides fault tolerance against the probability of failure of one or 
more LI servers. Fault tolerance in the system is further described in a section 
below entitled Fault Tolerance. 

5 

Fig. 4 schematically represents the chronology of RTM-mediated data 
flow between control cards 30A and SOB of router 10, according to one 
embodiment of the invention. Only two control cards are depicted in Fig. 4, 
however it is to be understood that the principles of data flow could also apply 

10 to a larger number of control cards. Control cards 30A and SOB run services A 
and B, respectively. Each control card SOA and SOB also has an RTM task 
running, RTM A, RTM B, respectively. The fact of each of the processors 
running a service task dictates that RTM A and RTM B are both Level-1 as 
defined hereinabove. Data flow is initiated by information injection from service 

15 A to RTM A, as indicated by arrow 1 . From RTM A, information is distributed 
concurrently to both route table A and to RTM B, as indicated by the two 
arrows each labeled 2. Thereafter, information is distributed from RTM B to 
route table B, as indicated by arrow 3. Finally, information is received by Service 
B fi-om route table B, arrow 4. Data flow of the type illustrated in Fig. 4 enables 

20 the timely distribution of routing database updates between a plurality of control 
cards within a distributed processor router, in which the plurality of control 
cards are jointly responsible for running a plurality of different services. 

By registration among Lis in the manner described herein, 
25 information generated by the full complement of services of the system can 
be effectively exchanged between Lis, with the result that each LI maintains 
a synchronized routing database. Scalability of the distribution of database 
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information among Lis is achieved by the formation of distribution trees 
during the registration process. 

According to the invention, each LI task will maintain a 
5 synchronized copy of the routing database. Each LI task has the role of 
constructing a synchronized forwarding table for use by L2 tasks, wherein 
the forwarding table consists of a subset of the routing database. 
Specifically, the routing database consists of all route entries, both best and 
non-best as defined above, as well as interface information. Each LI is able 
10 to construct the forwarding table, based on the routing database, by 
identifying the best route for each destination prefix. 

In this manner, when a best route is deleted fi-om the routing 
database, each LI can immediately replace the deleted "best route" with the 
15 next best route in the forwarding table which matches the particular 
destination prefix. 

An L2 task is an RTM task which is running on a processor outside 
of the LI pool. Each L2 requires a copy of the forwarding table. The 
20 source for forwarding table information are LI tasks that are running 
throughout the system. 

The hierarchical relationship of RTM tasks, according to a preferred 
embodiment of the invention, is schematically represented in Fig. 5. Lis 
25 represent the highest level, or top layer, of the hierarchical relationship. As 
described above, Lis are Level- 1 RTM tasks which maintain a synchronized 
copy of the routing database and are the source of the forwarding table, 
whereas L2s are Level-2 RTM tasks which only maintain a copy of the 
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forwarding table. L2s themselves can occupy different hierarchical levels. 
In order to distinguish between L2s which occupy different hierarchical 
levels, L2 nodes which are cHents of LI servers as well as servers of L2 
clients may be designated L2's; while L2s which are clients of L2' nodes may 
5 be designated L2"s. Thus, immediately below the Lis, at the intermediate 
hierarchical level or layer, lie L2s that are registered with Lis for forwarding 
table information. Below the intermediate hierarchical level lie L2's which 
are registered with an L2 node. Further, L2"s may be registered with L2's. 
According to a preferred embodiment, the depth of the topology shown in 

10 Fig. 5 is kept low by having a large fan-out at Layer 1 . Again with reference 
to Fig. 5, it should be noted that although only a single server is shown for 
each client, according to a currently preferred embodiment of the invention 
designed for fault tolerance, i.e. tolerance of the router system to failure of a 
RTM task server, each client has at least two servers. In practice, for a 

15 given L2" client (Layer 4), one server can be a Layer I server (LI), and the 
other can be a Layer 2 node. 

According to the invention, communication between RTM task 
chents and RTM task servers takes place to form scalable and fault tolerant 

20 distribution topologies. Among LI tasks, distribution trees are formed for 
the propagation of routing database information. An LI task which is 
running in association with a given service has the role of sourcing routing 
database information generated by that service. Distinct distribution trees 
therefore exist per service for the exchange of routing database information 

25 among LI tasks. In a similar manner, distribution trees for the propagation 
of the forwarding table are formed with LI tasks as the source of forwarding 
table information and L2 tasks as the nodes and leaves. 



19 



The RTM interacts with a Location Service module to determine the 
location of all RTM tasks running within router system 10. That is, the 
Location Service (LS) functions as a directory service. Interactions of the 
RTM with the LS include: (1) LI RTM tasks, running on a control card 30, 

5 query the LS to determine the location of any RTM tasks acting as the 

source of routing database information for a particular service; (2) L2 RTM 
tasks query the LS to determine the location of any LI RTM tasks (sources 
of forwarding table information); (3) LS notifies the RTM in the event that 
an RTM task comes up or goes down and (4) RTM tasks provide LS with 

10 RTM task type (including the routing database source) and level information 
to answer queries described in (1) through (3). 

As described above, Lis are responsible for propagating the 
forwarding database to the Level-2 tasks (L2s). This is accomplished by the 

1 5 establishment of L 1 -L2 client-server relationships. L2 nodes register with 
Lis for the forwarding table only (i.e., L2 nodes register for the forwarding 
table "service"). According to one aspect of the invention, an LI server will 
accept N L2 chents, where N is determined, at least in part, by the 
configured maximum fan-out. This situation is schematically represented in 

20 Fig. 6, in which an LI server (LI A) already has N L2 cUents, represented by 
L2A, L2B, and up to L2N. Client M represents an L2 that is not a client of 
an RTM task running in the control plane of the router system. If cUent M 
then signals a request to register v^th LI A (arrow 1), that request is denied 
as represented by arrow 2. If maximum fan-out has been reached on all Lis 

25 in the control plane, client M then requests registration (arrow 3) with an L2, 
e.g. L2A, that is a client of an LI (in this case Ll A), A registration response 
message is then sent from L2A to client M, as represented by arrow 4. 
Client M can now receive forwarding table updates from Ll A via L2A. 
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Maximum fan-out in L1-L2 client-server relationships is determined, inter 
alia, by CPU load. In case maximum fan-out of all L2 servers has been 
reached, then a client can force registration. This client-server registration 
procedure is used to form distribution trees for the propagation of the 
5 forwarding database among all L2 clients. Information on the location of the 
servers is available from the LS. According to a currently preferred 
embodiment, the LS itself runs on all control cards 30 and line cards 40 of 
router system 10. 

10 It will be apparent to the skilled artisan that the client-server 

registration procedure described here is hierarchically based, in that L2s first 
attempt to register with Lis until maximum fan-out has been reached, and 
only then will an L2 attempt to register with an L2 that is registered as a 
cHent of an Ll . An L2 which acts as a server to an L2 client may be 

15 designated L2', and an L2 cHent of an L2' server may be designated L2" (Fig. 
5). Large scale distribution is therefore achieved by using a reUable multicast 
transmission at the tree nodes. In general, the number of L2s is greater than 
the number of Lis. According to one embodiment, the ratio of Lis to L2s 
ranges from about 1:1 to about 1:15. 

20 

Fault Tolerance 

Fault tolerance in the system of the invention, as alluded to briefly 
above, is achieved by redundancy in registration, and therefore in 
25 communication. As a client, an Ll or L2 task registers with at least two 
servers from which it may receive the same information. One of the servers 
with which the client registers is considered a primary server, and the other 
a secondary. The client communicates exclusively with the primary unless 
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and until the primary fails in some manner to deliver, and then the client 
turns to the secondary for database updates. Service is thus uninterrupted. 

In the event of a server failure, and a necessary switchover by a 
5 client to its secondary server, the client receives a copy of the secondary 
server's database. If the client is a node in a distribution tree, it simply 
delivers to its clients the difference between the existing database and the 
copy of the database received from the secondary server. 

10 Referring now to Fig. 7A, the role of a control card as a Level-2 

node is to receive forwarding entries from its primary LI server, and then to 
redistribute the forwarding entries to its own clients, represented as L2 
clients A, B, and C, The L2 node is registered with two LI servers, the 
primary LI server and the secondary LI server, for the purpose of fault 

15 tolerance, as schematically represented in Fig. 7 A. 

Referring now to Fig. 7B, if the primary LI server fails, the L2 node 
activates its secondary LI server. When the secondary LI server is activated, 
it delivers a complete copy of its database to the L2 node, as schematically 
20 represented in Fig. 7B. When the L2 node receives the copy of the entire 
table from the secondary LI server, it compares that copy to its existing 
database, and calculates the difference between the two. It only needs to 
distribute to L2 clients A, B and C the difference between the entire new 
table and its existing table. 

25 

Fig. 8 schematically represents a series of steps in a method for the 
synchronized distribution of routing data within a distributed processor, 
highly-scalable router, according to one embodiment of the invention. Step 
800 of Fig. 8 involves running at least one routing protocol of a complement 
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of routing protocols on individual ones of a first plurality of processors, 
wherein each routing protocol of the complement of routing protocols 
generates routing data. This first plurality of processors are the LI 
processors described in detail above. Also as previously described, it is the 
5 configuration of the Lis to run routing protocols and to otherwise behave as 
Lis that makes them Lis. An LI may not be running a routing protocol, but 
still be an LI . That is, an LI may obtain all of its routing data from other 
Lis with which it registers as a cUent. 

Step 802 involves registering each of the first plurality of processors 

10 with at least one other of the first plurality of processors. Step 804 involves 
exchanging the routing data between members of the first plurality of 
processors, such that each of the first plurality of processors receives a fiiU 
complement of routing data generated by the complement of routing 
protocols. The complement of routing data received by each of the first 

15 plurality of processors provides a complete routing database. Step 806 

involves forming a forwarding database from the complete routing database 
provided as a result of step 804. The forwarding database formed in step 
806 is comprised of a subset of the complete routing database provided in 
step 804, 

20 Step 808 involves propagating the forwarding database from the first 

plurality of processors to a second plurality of processors of the distributed 
processor router, wherein the second plurality of processors are 
characterized as not running (or being configured to run) routing protocols. 
The method steps 800 through 808 may be sequentially repeated over time, 

25 for example, when updated reachability information is received from one or 
more peer routers of the distributed processor router. 
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General Applicability 



The embodiments of the present invention described in enabling 
detail above have all been related to routing of data packets in multi- 
5 processor, scalable routers. These embodiments are exemplary, and 
represent a preferred application of the new technology described, but are 
not limiting in applicability of the invention. There are numerous other 
situations and systems in which the apparatus and methods of the invention 
may provide advantages. These situations include all situations in which 
10 multiple processors may be employed in parallel processing applications, 
wherein maintenance of one or more common databases is the object. The 
description of the present invention is intended to be illustrative, and not to 
limit the scope of the appended claims. Many alternatives, modifications, 
and variations will be apparent to those skilled in the art. 
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WHAT IS CLAIMED IS 



1 . In a distributed processor system wherein a first and a second protocol 
operating on individual ones of a first plurality of processors are involved in 

5 independently generating or amending data for a single database, and 
wherein each of the first plurality of processors maintains a copy of the 
database, a method for synchronized maintenance and distribution of the 
database, comprising the steps of 

(a) registering each of the first plurality of processors with at least 
10 one other of the first plurality of processors, creating client-server pairs, in 

an arrangement that each of the plurality of processors either runs or is 
registered with a processor running the first and second protocols; and 

(b) sharing the generated or amended data from the servers to the 
registered chents, such that each of the first plurality of processors receives 

15 generated or amended data from both the first and second protocols. 

2. The method of claim 1 wherein the system comprises a second plurality 
of processors upon which the first and second protocol do not run, and 
fiirther comprising a step (c) for registering each of the second plurality of 

20 processors with at least one of the first plurality of processors, creating 

client-server pairs between individual ones of the first and second plurality of 
processors, and a step (d) for sharing at least a subset of the database from 
the servers in the first plurality of processors to the clients in the second 
plurality of processors. 

25 

3. The method of claim 2 comprising a third plurality of processors upon 
which the protocols do not run, and fiirther comprising a step (e) for 
registering each of the third plurality of processors with individual ones of 
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the second plurality of processors, creating client-server pairs between 
individual ones of the second and third plurality of processors, enabling 
clients in the third plurality of processors to receive copies of the subset of 
the database. 

5 

4. The method of claim 3 wherein, in one or more of steps (a), (c) and (e) 
clients register with a second processor to create a redundant server-cUent 
relationship for fault tolerance. 

10 5. The method of claim 4 wherein a client treats the two servers with which 
it registers as a primary and a secondary server, and communicates only with 
the primary server as long as the primary server remains capable, and further 
comprising a step for activating the secondary server in the event the primary 
server fails. 
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6- The method of claim 5 wherein, upon activation of the second server, a 
copy of the database is sent to the client, which compares that copy with its 
own copy, determines the difference, and uses only the difference in further 
propagation of copies. 



7. A distributed processor system comprising: 

a first plurality of processors, each maintaining a copy of a single 
database; and 

a first and a second protocol operating on individual ones of the first 
25 plurality of processors, the protocols independently generating or amending 
data for the single database; 

characterized in that each of the first plurality of processors registers 
with at least one other of the first plurality of processors, creating chent- 
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server pairs, in an arrangement that each of the plurality of processors either 
runs or is registered with a processor running the first and second protocols, 
and the servers of the client-server pairs share the generated or amended 
data with the chents, such that each of the first plurality of processors 
5 receives generated or amended data from both the first and second 
protocols. 

8. The system of claim 7 comprising a second plurality of processors upon 
which the first and second protocol do not run, wherein each of the second 

10 plurality of processors registers with at least one of the first plurality of 
processors, creating client-server pairs between individual ones of the first 
and second plurality of processors, and at least a subset of the database is 
shared from the servers in the first plurality of processors to the clients in the 
second plurality of processors. 

15 

9. The system of claim 8 comprising a third plurality of processors upon 
which the protocols do not run, wherein each of the third plurality of 
processors register with individual ones of the second plurality of 
processors, creating client-server pairs between individual ones of the second 

20 and third plurality of processors, enabling clients in the third plurality of 
processors to receive copies of the subset of the database. 

10. The system of claim 9 wherein clients register with a second processor 
to create a redundant server-client relationship for fault tolerance. 

25 

11. The system of claim 10 wherein a client treats the two servers with 
which it registers as a primary and a secondary server, communicates only 
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with the primary server as long as the primary server remains capable, and 
activates the secondary server in the event the primary server fails. 

12. The system of claim 1 1 wherein, upon activation of the second server, a 
5 copy of the database is sent to the client, which compares that copy with its 

own copy, determines the difference, and uses only the difference in further 
propagation of copies. 

13. In a data packet router wherein a first and a second routing protocol 
10 generating routing data operate on individual ones of a first plurality of 

processors, and wherein each of the first plurality of processors maintains a 
copy of the routing table, a method for synchronized maintenance and 
distribution of the routing table and a forwarding table subset of the routing 
table, comprising the steps of 
15 (a) registering each of the first plurality of processors with at least 

one other of the first plurality of processors, creating cUent-server pairs, in 
an arrangement that each of the plurality of processors either runs or is 
registered with a processor running the first and second routing protocols; 
and 

20 (b) sharing the routing data firom the servers to the registered clients, 

such that each of the first plurality of processors receives the routing data 
from both the first and second routing protocols. 

14. The method of claim 13 wherein the data packet router comprises a 
25 second plurality of processors upon which the first and second protocol do 

not run, and further comprising a step (c) for registering each of the second 
plurality of processors with at least one of the first plurality of processors, 
creating client-server pairs between individual ones of the first and second 
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plurality of processors, and a step (d) for sharing a forwarding table subset 
of the routing table from the servers in the first plurality of processors to the 
clients in the second plurality of processors. 

5 15. The method of claim 14 comprising a third plurality of processors upon 
which the protocols do not run, and further comprising a step (e) for 
registering each of the third plurality of processors with individual ones of 
the second plurality of processors, creating client-server pairs between 
individual ones of the second and third plurality of processors, enabling 
10 clients in the third plurality of processors to receive copies of the forwarding 
table. 

16. The method of claim 15 wherein, in one or more of steps (a), (c) and (e) 
clients register with a second processor to create a redundant server-client 

15 relationship for fault tolerance. 

17. The method of claim 16 wherein a client treats the two servers with 
which it registers as a primary and a secondary server, and communicates 
only with the primary server as long as the primary server remains capable, 

20 and further comprising a step for activating the secondary server in the event 
the primary server fails. 

18. The method of claim 17 wherein, upon activation of the second server, a 
copy of the routing table of forwarding table is sent to the client, which 

25 compares that copy with its own copy, determines the difference, and uses 
only the difference in further propagation of copies, 

19. A data packet router comprising: 
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a first plurality of processors, each maintaining a copy of a routing 
table; and 

a first and a second protocol operating on individual ones of the first 
plurality of processors, the protocols independently generating or amending 
5 routing data for the routing table; 

characterized in that each of the first plurality of processors registers 
Avith at least one other of the first plurality of processors, creating client- 
server pairs, in an arrangement that each of the plurality of processors either 
runs or is registered with a processor running the first and second routing 
10 protocols, and the servers of the client- server pairs share the generated or 
amended routing data with the clients, such that each of the first plurality of 
processors receives generated or amended routing data from both the first 
and second routing protocols. 

15 20. The router of claim 19 comprising a second plurality of processors upon 
which the first and second routing protocols do not run, wherein each of the 
second plurality of processors registers with at least one of the first plurality 
of processors, creating client-server pairs between individual ones of the first 
and second plurality of processors, and at least a forwarding table subset of 

20 the routing table is shared from the servers in the first plurality of processors 
to the chents in the second plurality of processors. 

21 , The router of claim 20 comprising a third plurality of processors upon 
which the routing protocols do not run, wherein each of the third plurality of 
25 processors register with individual ones of the second plurality of 

processors, creating chent-server pairs between individual ones of the second 
and third plurality of processors, enabling clients in the third plurality of 
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processors to receive copies of the forwarding subset of the routing 
database. 

22. The router of claim 20 wherein chents register with a second processor 
5 to create a redundant server-chent relationship for fault tolerance. 

23. The router of claim 22 wherein a chent treats the two servers with 
which it registers as a primary and a secondary server, communicates only 
with the primary server as long as the primary server remains capable, and 

10 activates the secondary server in the event the primary server fails. 

24. The router of claim 23 wherein, upon activation of the second server, a 
copy of the routing or forwarding table is sent to the chent, which compares 
that copy with its own copy, determines the difference, and uses only the 

1 5 difference in further propagation of copies. 



31 



ABSTRACT OF THE DISCLOSURE 



A method of and apparatus for distributing data for a database between a 
5 plurality of processors in a distributed processor system involves running a database 
management system on a first plurality of processors in conjunction with a plurality 
of protocols that generate or amend data for the database. Data is distributed from 
servers to clients registered in a server-client relationship. Server-client relationships 
may also be registered between a second, and a third plurality of processors that do 
10 not run protocols generating or amending data for the database. Fault tolerant 
redundancy is provided by clients registering with two or more servers, one as a 
primary and another as a secondary, and activating the secondary if the primary fails. 
The method is particularly applicable to scalable data packet routers having a 
plurality of processors operating on different line and control cards. 
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and, insofar as the subject matter of each of the claims of this application is not disclosed in the prior United States 
application in the manner provided by the first paragraph of Title 35, United States Code, si 12, 1 acknowledge the duty 
to disclose material information as defined in Title 37, Code of Federal Regulations, si 56(a) which occurred between 
the filing date of the prior application and the national or PCT international filing date of this application. 



(Application Serial No.): (Filing Date): (Status): 

(Application Serial No.): (Filing Date): (Status): 

(Application Serial No.): (Filing Date): (Status): 

(Application Serial No.): (Filing Date): (Status): 

(Application Serial No.): (Filing Date): (Status): 



POWER OF ATTORNEY: As a named inventor, I hereby appoint the following attomey(s) and/or agent(s) to 
prosecute this application and transact all business in the Patent and Trademark Office connected therewith. 
(List name and registration number) 

Name:Donald R. Boys Reg. No. 35,074 



SEND CORRESPONDENCE TO: 
Donald R. Boys 
P.O. Box 187 
Aromas, CA 95004 



DIRECT TELEPHONE CALLS TO: 
Donald R. Boys (831) 726-1457 



I hereby declare that all statements made herein of my own knowledge are true and that all statements made on 
information and belief are believed to be true; and further that these statements were made with the knowledge that 
willful false statements and the like so made are punishable by fine or imprisonment, or both, under Section 1001 of 
Title 18 of the United States Code and that such willful false statements may jeopardize the validity of the application 
or any patent issued thereon. 



Full name of sole or first inventor: Puneet Agarwal 

1st inventor's signatuT^!= ^^. ^^^^^^^"^ -^^^ 

Residence: 50 Chumasero Drive. #10M. San FranciscoYCA 94m - Citizenship: India 
Post Office Address: Same 

Full name of 2nd joint inventopf^pf any: Russell R. Tuck. Ill . 

2nd inventor's signature: ^U<^J^/C ^^^>L^^ Dated: _/^/*^^ 

Residence: 1136 S. Blanev Ave.. San Jose. CA. 95129 Citizenship: US 
Post Office Address: Same 

Full name of 3rd joint inventor, if any: Bora Avdin Akvol ^ 

3rdinventor-ssi^ature: ^ <^ Dated: bljlj OO 

Residence: 4609 La Crescenif Loop. San Jose,t€fA, 95136 " ^itizenship: Turkey 
Post Office Address: Same 

Full name of 4th joint inventor, if any: Erol Basturk / 

4th inventor's .ionnture: fni ^ njAntw^ Dated: ij/j ^ 

Residence: 10246 Will Court Cupertino. C^. 95014 Citizenship: Switzerland 
Post Office Address: Same 

Full name of 5th joint inventor, if any: Mike Mussohne 

5th inventor's signature: T^ffJ^/ T^iM^T^"^ D^it±tlll^ 

Residence: 620 Iris Ave.. #14. Sunnwale. CA 94086 Citizenship: US 
Post Office Address: 



Full name of 6th joint inventor, if any: 

6th inventor's signature: 

Residence: Citizenship: 

Post Office Address: 

Full name of 7th joint inventor, if any: 

7th inventor's signature: 

Residence: Citizenship: 

Post Office Address: 



Full name of 8th joint inventor, if any: . 



8th inventor's signature: . Dated: 

Residence: Citizenship: 

Post Office Address: 
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